Add support for Calculating Risk in VulnerableCode #1593

ziadhany · 2024-09-17T02:15:26Z

vulnerabilities/risk.py

DennisClark · 2024-10-21T22:54:21Z

it would be really great to see a review of this PR soon!

Set data_source as the header for the exploit table. Squash the migration files into a single file. Add test for exploit-db , metasploit Add a missing migration file Rename resources_and_notes to notes Fix Api test Refactor metasploit , exploitdb , kev improver Rename Kev tab to exploit tab Add support for exploitdb , metasploit, kev Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Add pipeline_id for ( kev, metasploit, exploit-db ) Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Set data_source as the header for the exploit table. Squash the migration files into a single file. Add test for exploit-db , metasploit Add a missing migration file Rename resources_and_notes to notes Fix Api test Refactor metasploit , exploitdb , kev improver Rename Kev tab to exploit tab Add support for exploitdb , metasploit, kev Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

uncomment all importers Signed-off-by: ziad hany <[email protected]>

TG1999 · 2024-10-28T08:53:47Z

@ziadhany this generally looks good to me, can we have some logs for the pipeline and some UI and API screenshots so we can merge it.

keshav-space

Thanks @ziadhany, some nits for your consideration.
Also, please use LoopProgress while iterating over large records, as we do here

vulnerablecode/vulnerabilities/pipelines/enhance_with_exploitdb.py

Line 61 in 590c91a

    
           progress = LoopProgress(total_iterations=fetched_exploit_count, logger=self.log)

vulnerabilities/pipelines/risk_package.py

vulnerabilities/risk.py

vulnerabilities/pipelines/risk_package.py

vulnerabilities/risk.py

vulnerabilities/models.py

vulnerabilities/migrations/0074_package_risk.py

vulnerabilities/risk.py

…skPackagePipeline to ComputePackageRiskPipeline. Add a tooltip for risk, and remove any unused imports in the view. Signed-off-by: ziad hany <[email protected]>

…ore and remove any extra whitespace in views.py. Signed-off-by: ziad hany <[email protected]>

Signed-off-by: ziad hany <[email protected]>

ziadhany · 2024-10-28T15:30:48Z

@TG1999 @keshav-space
Thank you for reviewing this PR. Here are the pipeline logs:

/home/ziad-hany/PycharmProjects/vulnerablecode/venv/bin/python /home/ziad-hany/PycharmProjects/vulnerablecode/manage.py improve localhost:8000 --all 
Improving data using compute_package_risk
INFO 2024-10-28 13:10:07.074 Pipeline [ComputePackageRiskPipeline] starting
INFO 2024-10-28 13:10:07.074 Step [add_risk_package] starting
INFO 2024-10-28 13:10:07.755 Calculating risk for 84,396 affected package records
INFO 2024-10-28 13:14:57.360 Progress: 10% (8440/84396) ETA: 2601 seconds (43.4 minutes)
INFO 2024-10-28 13:17:30.519 Progress: 20% (16880/84396) ETA: 1768 seconds (29.5 minutes)
INFO 2024-10-28 13:19:28.477 Progress: 30% (25319/84396) ETA: 1307 seconds (21.8 minutes)
INFO 2024-10-28 13:22:06.047 Progress: 40% (33759/84396) ETA: 1076 seconds (17.9 minutes)
INFO 2024-10-28 13:25:15.062 Progress: 50% (42198/84396) ETA: 907 seconds (15.1 minutes)
INFO 2024-10-28 13:30:57.706 Progress: 60% (50638/84396) ETA: 833 seconds (13.9 minutes)
INFO 2024-10-28 13:36:04.109 Progress: 70% (59078/84396) ETA: 667 seconds (11.1 minutes)
INFO 2024-10-28 13:39:06.350 Progress: 80% (67517/84396) ETA: 434 seconds (7.2 minutes)
INFO 2024-10-28 13:42:57.214 Progress: 90% (75957/84396) ETA: 219 seconds (3.6 minutes)
INFO 2024-10-28 13:48:50.555 Progress: 100% (84396/84396)
INFO 2024-10-28 13:48:50.694 Successfully added risk score for 84,396 package
INFO 2024-10-28 13:48:50.721 Step [add_risk_package] completed in 2324 seconds (38.7 minutes)
INFO 2024-10-28 13:48:50.721 Pipeline completed in 2324 seconds (38.7 minutes)

Process finished with exit code 0

tdruez · 2024-10-29T06:48:25Z

@ziadhany @TG1999 Could we make a push on this one as this is a key addition for the VEX support in DejaCode.
Also, make sure it is available everywhere in the relevant API endpoint as we need to capture this data.

vulnerabilities/models.py

vulnerabilities/pipelines/risk_package.py

keshav-space

Thanks @ziadhany, see some suggestion.
Also, please rename the pipeline file from risk_package.py to compute_package_risk.py and the corresponding test file to test_compute_package_risk.py.

vulnerabilities/pipelines/risk_package.py

Add pagination and refactor bulk_update_package Signed-off-by: ziad hany <[email protected]>

ziadhany · 2024-10-29T13:07:35Z

@tdruez @keshav-space, I've completed the requested changes and I hope we can move forward and merge this PR.

Additionally, please review the weight_config.json file to ensure an accurate view of the risk. The default weight is set to one and may need adjustment to reflect typical risk values if we plan to use it in Dejacode.

keshav-space

Thanks @ziadhany, looking good! I ran the pipeline locally and it worked alright.
Before we merge this, let's discuss the weight configuration in our community call today.
Also, we will need to revisit the risk computation once #1597 is resolved.

DennisClark · 2024-11-05T16:17:25Z

note that @keshav-space is going to create a list of sources to expand the entries in weight-config.json and we'll take it from there. Please put suggested weights in the file. We probably also need a default.

ziadhany force-pushed the risk branch from 35eb0a4 to 49bf7e3 Compare October 1, 2024 00:14

ziadhany marked this pull request as ready for review October 1, 2024 00:20

ziadhany requested review from TG1999 and keshav-space October 1, 2024 14:48

TG1999 reviewed Oct 15, 2024

View reviewed changes

vulnerabilities/risk.py Show resolved Hide resolved

TG1999 added the 2-next label Oct 15, 2024

ziadhany force-pushed the risk branch from 61687c8 to 8d9f6d8 Compare October 22, 2024 13:11

ziadhany added 11 commits October 22, 2024 16:13

Implement the appropriate LoopProgress progress bar.

0dbe640

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Fix migration conflict

f5131b7

Add pipeline_id for ( kev, metasploit, exploit-db ) Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Implement the appropriate LoopProgress progress bar.

4b808d9

Refactor the error handling logic in the code. Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Remove unwanted migration file

6e67497

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Add support for Calculating Risk in VulnerableCode

2d9e585

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Remove unwanted migration file

22cdb76

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Add a prefetch to try to optimize query performance

8beb3b9

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Empty risk when there is no data

74a16ff

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

Create a pipeline for package risk

8a637fc

Signed-off-by: ziadhany <[email protected]> Signed-off-by: ziad hany <[email protected]>

ziadhany force-pushed the risk branch from 8d9f6d8 to 8a637fc Compare October 22, 2024 13:17

Load the weight once

848a5f5

uncomment all importers Signed-off-by: ziad hany <[email protected]>

ziadhany requested a review from TG1999 October 22, 2024 13:40

Merge branch 'main' into risk

02b7cf2

keshav-space requested changes Oct 28, 2024

View reviewed changes

ziadhany added 5 commits October 28, 2024 17:46

Update the risk description in the model. Rename the pipeline from Ri…

185eb0e

…skPackagePipeline to ComputePackageRiskPipeline. Add a tooltip for risk, and remove any unused imports in the view. Signed-off-by: ziad hany <[email protected]>

Merge remote-tracking branch 'origin/risk' into risk

91c2c95

Rename the pipeline step from add_risk_package to add_package_risk_sc…

45c62b5

…ore and remove any extra whitespace in views.py. Signed-off-by: ziad hany <[email protected]>

Resolve migration conflict

0af9432

Signed-off-by: ziad hany <[email protected]>

Merge remote-tracking branch 'origin/main' into risk

d9fb46b

ziadhany force-pushed the risk branch from 4ac55eb to d9fb46b Compare October 28, 2024 15:23

ziadhany requested a review from keshav-space October 29, 2024 05:38

tdruez requested changes Oct 29, 2024

View reviewed changes

vulnerabilities/models.py Outdated Show resolved Hide resolved

vulnerabilities/pipelines/risk_package.py Outdated Show resolved Hide resolved

keshav-space requested changes Oct 29, 2024

View reviewed changes

vulnerabilities/pipelines/risk_package.py Outdated Show resolved Hide resolved

vulnerabilities/pipelines/risk_package.py Outdated Show resolved Hide resolved

vulnerabilities/pipelines/risk_package.py Outdated Show resolved Hide resolved

Rename the pipeline file

0e114ac

Add pagination and refactor bulk_update_package Signed-off-by: ziad hany <[email protected]>

ziadhany requested review from tdruez and keshav-space October 29, 2024 13:08

pombredanne assigned pombredanne and TG1999 and unassigned pombredanne Oct 31, 2024

pombredanne mentioned this pull request Oct 31, 2024

Incorrect severity score due to identical Reference URLs #1597

Open

keshav-space approved these changes Nov 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Calculating Risk in VulnerableCode #1593

Add support for Calculating Risk in VulnerableCode #1593

ziadhany commented Sep 17, 2024 •

edited

Loading

DennisClark commented Oct 21, 2024

TG1999 commented Oct 28, 2024

keshav-space left a comment

ziadhany commented Oct 28, 2024

tdruez commented Oct 29, 2024

keshav-space left a comment

ziadhany commented Oct 29, 2024

keshav-space left a comment

DennisClark commented Nov 5, 2024 •

edited

Loading

Add support for Calculating Risk in VulnerableCode #1593

Are you sure you want to change the base?

Add support for Calculating Risk in VulnerableCode #1593

Conversation

ziadhany commented Sep 17, 2024 • edited Loading

DennisClark commented Oct 21, 2024

TG1999 commented Oct 28, 2024

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Oct 28, 2024

tdruez commented Oct 29, 2024

keshav-space left a comment

Choose a reason for hiding this comment

ziadhany commented Oct 29, 2024

keshav-space left a comment

Choose a reason for hiding this comment

DennisClark commented Nov 5, 2024 • edited Loading

ziadhany commented Sep 17, 2024 •

edited

Loading

DennisClark commented Nov 5, 2024 •

edited

Loading